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Abstract 
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1 Preliminaries 

1. "Probability Distributions," Connexions module m43336, 

2. "Introduction to Markov Processes," Connexions module m44014, 

3. "Continuous Markov Processes," Connexions module m44258 (in parallel with "Integral of a Markov 
Process," module m44376), 

4. "Integral of a Markov Process," Connexions module m44376 (in parallel with "Continuous Markov 
Processes," module m44258). 

2 Introduction 

A jump Markov process proceeds by the Markov system performing finite, that is, not infinitesimal jumps 
on moving from x (t) to x(t + dt). Although we will be able to demonstrate that the Markov propagator 
density function for such a process can be fully characterized by two functions, as was also the case with the 
continuous Markov processes, on account of this finiteness we will no longer be able to truncate Kramers- 
Moyal equations. There is no Fokker-Planck equivalent for the jump Markov process. 

Because the Kramers-Moyal equations don't truncate, there is another machinery, the master equations, 
which are differential-integral equations, that is preferably used in this context. Although still intractable 
analytically, the master equations provide us with tools for numerical investigations of jump processes. 

The jump Markov process propagator density looks as follow 

n (x \dt\ (x, t)) = a (x, t) dt w (x' | (x, t)) + (1 — a (x, t) dt) 6 (x ) , (1) 

where 

• a (x, t) dt is the probability that the Markov process at x at time t will jump within the next dt, that 
is, it will jump during the time interval [t, dt] and 

• w (x'| (x, t)) dx is the probability that having jumped the process will land between x' and x' + dx 
away from x, that is between x + x' and x + x' + dx . 

♦Version 1.15: Nov 8, 2012 3:49 pm -0600 
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The same propagator density is sometimes expressed as 

n(x|dt|(x,t)) = 

W(x'\(x,t)) dt + (l-/ n(x ")W(x"|(x,t)) dtdx")j(x) ) 

where W (x'| (x, t)) is called the consolidated characterizing function of a jump Markov process. It is the 
probability that the Markov process at x at t will jump in the next dt to between x' and x' + dx away from 
x. Functions a (x, t) and w (x'| (x, t)) can be expressed in terms of W (x'| (x,t)) as follows 

= /n(*-) W ( x 'K x '*)) dx '> . , 

w f x ' I fx ^ - ^(*K*.*)) ( 3 ) 

W\X\{X,t)) - /n(x „ )M / (x "| (Xit))dx "- 

The master equations are expressed in terms of the consolidated characterizing functions and look as follows 
(forward and backward): 

J>P((x,i)|(x ,i )) = 
Jn ( x ) ( W (x'| (x - x, t)) P ((x - x, t) | (x , ^)) (4) 
- W(-x'\(x,t))P((x,t)\(x ,t )) ) 

-4P((x,i)|(x ,i )) = 

/nCx-j^KK^^o)) (P((x,t)| (xo-x',to)) -P((x,t)|(xo,t ))) 

Unlike continuous Markov processes, the jump processes can be characterized by a yet another function, 
called the next-jump density function 

p((x',t) | (x,i)) dt dx. (6) 

The function is the probability that a jump Markov process that is at x at t will perform its next jump 
between t + t' and t + t ' + dt' landing between x + x and x + x + dx , or, to put it in other words, that the 
next jump will happen within [t' ,t' + dt']aftert landing the system within [x',x' + dx'jaway from x. This 
function can be expressed in terms of a(x, t) and w (x'| (x,i)) as follows 

p ((x\ t) | (x, t)) = a (x, t + t) e~ So a ( x ' t+t ") dt "w (x'| (x, t + t')). (7) 
This will simplify for temporally homogeneous processes to 

p((x',t) | (x,t)) =a(x) e- a ( x ) t 'w;(x'|x). (8) 

Completely homogeneous jump Markov processes are particularly interesting, because they're the simplest 
possible and so we can say something about them. The two characterizing functions, a (x, t) and w (x'\ (x, t)) 
become 

a ( x ^) =a, ^ 
w(x'\(x,t)) =w(x), 

with the next jump density function simplifying to 

p((x',t) | (x,t)) = aw (x) e- at ' . (10) 

In this context it is interesting to consider w (x ) given by exponential, Gaussian and Cauchy-Lorentz 
distributions. The systems so defined become quite tractable and, more importantly, applicable to a range 
of physical phenomena, for example, diffusion and Brownian motion. 
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In this module we are also going to take another look at quantum mechanics, asking if quantum mechanics 
can be simulated by jump Markov processes. There was a vigorous discussion about this in the literature. 

It all started with a paper by Hardy, Home, Squires and Whitaker, "Realism and the quantum-mechanical 
two-state oscillator," published on the 1st of April 1992 in Physical Review A, vol. 45, no. 7, pp. 4267-4270. 
Nearly two years later Gillespie referred to this paper in his article "Why quantum mechanics cannot be 
formulated as a Markov process," published in March 1994 in Physical Review A, vol. 49, no. 3, pp. 1607- 
1612. A year after that, in May 1995, Garbaczewski and Olkiewicz responded with "Why quantum dynamics 
can be formulated as a Markov process," published in Physical Review A, vol. 51, no. 5, pp. 3445-3453, 
to which Gillespie responded in June 1996 in "Comment on 'Why quantum dynamics can be formulated 
as a Markov process'", published in Physical Review A, vol. 53, no. 6, pp. 4602-4604. Not to be outdone, 
Gorbaczewski and Olkiewicz published their riposte in August 1996, "Comment on Why quantum mechanics 
cannot be formulated as a Markov Process'" in Physical Review A, vol. 54, no. 2, pp. 1733-1736. In the 
same issue of Physical Review A, following the article by Gorbaczewski and Olkiewicz, Gillespie presented 
his final one-page argument, pp. 1737-1738. But the last word in this exchange belonged to Hardy, Home, 
Squires and Whitaker, who in their "Comment on Why quantum mechanics cannot be formulated as a 
Markov process'", published in October 1997 in Physical Review A, vol. 56, no. 4, pp. 3301-3303, pointed 
out that they did not claim their process described in the original 1992 paper to be a Markov process. They 
demonstrated a concrete model that satisfied their requirements (of 1992) and pointed out various problems 
with Gillespie's own arguments. 

The important moral of the story is that not every stochastic jump process must be a Markov process to 
begin with. Markovianism is a quite special requirement that may not always apply. Another moral is that 
some reasoning that Gillespie is so fond of in his papers and in his book (which here we faithfully follow, 
because it's an excellent introduction to Markov processes) is not necessarily unquestionable. In particular, 
the trick of dividing dt into dt/n does stretch things a bit. Shouldn't we treat dt as indivisible instead? In 
effect the Gillespie's trick leads to smoothness conditions that may be stronger than necessary. Gillespie's 
dt is not infinitesimal enough. 

To understand this fascinating discussion though we must master the formalism of jump Markov processes 
first, and so we begin... 

3 The Jump Propagator 

A jump process found at x at t will most likely stay at x for a while, then it'll suddenly jump away from 
x. We can't normally say how long it's going to stay at x, but we are interested in processes for which a 
probability exists that the jump will occur within the next At. And so 



is this probability. The probability does not have to exist, but if it does not, then there's not much else 
we can say about such processes. So we stick to this assumption. There's one other thing we can obviously 
assume, namely that g(0| (x,i)) = 0, meaning that the process is stuck at x at t. We will also assume 
that (At | (x,t)) is a smooth function of times t and At. These two assumptions, in combination with the 
assumption that q exists in the first place, go quite far already. Whether we assume too much by doing so 
will transpire later, when we attempt to apply the theory to known physical processes. But for the time 
being, the assumptions are purely operational: we need them so that we can develop a tractable theory in 
the first place. It will turn out eventually that the assumption of Markovianism demands a certain degree 
of smoothness of q. 

Now, once the process has jumped, it'll land somewhere, and where it lands may be described by another 
probability 



«(At|(x,t)) 



(11) 



w 



(x | (x, t)) dx . 



(12) 
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This is the probability that once the process has jumped it'll land at between x' and x' + dx away from x, 
that is, between x + x' and x + x + dx'. And here we assume also that w (x'| (x,i)) is a smooth function 
of time t, again so that the theory remains tractable, but it'll transpire down the road that the assumption 
of Markovianism demands a certain degree of smoothness of w. 

Both functions q and w dx must be non-negative, because they are probabilities. Additionally w dx 
being probability density must integrate to 1. 

Theorem 3.1 

If At = dt is infinitesimal, then 

q(dt\(x,t)) = a(x,t) dt. (13) 

The proof of this equation is rooted in the Markov definition of the underlying process and in the proportional 
function lemma, that was proved in module m44258: 

Lemma 3.1 (Proportional Function) 

If 

1. / (x) is smooth, 

2. Vn e N : / (x) — n(f (x/n)) , where N is the set of natural numbers (positive integers), 

then / (x) — ax, where a does not depend on x (but it may depend on other parameters of the problem. 
We prove (13) as follows. We are going to demonstrate that 

q(dt\(x,t)) = nq(^\(x,t)y (14) 

from which (13) follows by the use of the Proportional Function lemma. To get there we divide the 
infinitesimal dt into n portions of length dt/n each. If q(dt\ (x, i)) is the probability that the system that's 
in x at t will jump away from x in the next dt, then 1 — q (dt\ (x,£)) is the probability that the system will 
not jump in the next dt. Since we have divided dt into n segments, the probability that the system will not 
jump in dt must be equal to the product of probabilities that the system will not jump in each time segment, 
that is 

1 -q(dt\ (x,i)) = f[ (l - 9 I (x,*i-i))) • (15) 

Alas, because dt is supposed to be an infinitesimal, the times tj_i are infinitesimally close to each other, so 
we can replace them all with just t, which yields 

l-q(dt\(x,t))= (l-g^|(x,t))) . (16) 

Now we make a yet another use of the fact that dt is infinitesimal. Let us recall that q(0\ (x, f)) = and 
since q is smooth, we can readily assume that q( — \ (x, t)) is infinitesimal too. Therefore we can approximate 

l-g^Kx,*))^ «l-ng(^|(x,t)), (17) 

wherefrom, in combination with (16), (14) follows, which ends the proof. 

In view of our comments on the discussion between Gillespie, Hardy and his collaborators, and Gor- 
baczewski and Olkiewicz, it is important that we recognize that, in this case, we have assumed the smooth- 
ness of q from the beginning. Whether this assumption is more than is strictly required by Markovianism 
remains to be seen. The reasoning here is quite similar to how we demonstrated in m44258 that 

< x (dt, (x,t)) > = fj,(x,t) dt, ^ 
var (x (dt,(x,t))\ =D(x,t)dt 
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for the continuous Markov processes, wherefrom we deduced that the propagator density function in this 
case had to be a Gaussian. The assumptions behind (13) preclude, for example, a sharp spike in probability 
q at the end of dt, or indeed, within a finite time interval At thereafter. 

Given the meaning of q, the meaning of a (x, t) dt is the probability that the process in x at t will jump 
away from x in the next dt. Furthermore, because dt is infinitesimal, we expect only one jump to occur 
within dt or none at all. The probability of two jumps occurring would be proportional to (dt) 2 . 

The jump Markov process propagator density function can now be written in the following form 

n (x'|cft| (x,t)) dx = 

, , ( 19 ) 

a (x, t) dtw (x'| (x, t)) dx + (1 — a (x, t) dt) 5 (x') dx', 

which we read as follows. On the left side we have the probability of the Markov process finding itself 
removed by x' from x which it occupied at t upon having advanced the time by dt. On the right side of the 
equation, we express the same as the probability that the process in x at t will jump in the next dt times the 
probability that having jumped it will land by x' away from xor — and this is what the plus stands for — the 
probability that the process will not jump, in which case it will stay at x, which is to say, it will displace by 
x = 0, which is what is signified here by the delta function. So, the process will jump, or it will not. 
Dividing both sides by dx yields 

n (x' |cft I (x,t)) = a(x,t) dtw (x'| (x,t)) + (1 - a(x,t) dt) S (x) . (20) 

In the following, we're going to look more closely at two issues. First, we'll derive a formula for q (At\ (x, t)) 
in terms of a(x,t). Second, we're going to see that (20) satisfies the Chapman-Komogorov equation to the 
first order in dt, wherefrom we'll also obtain a more precise idea as to how smooth the functions a (x, t) and 
w (x'| (x, t)) have to be. 

Expression for q(At\ (x, t)): We cannot infer from (13) that 

/■At 

q(At\ (x,t)) = / a(x,t) dt (21) 
Jo 

It is more tricky, because we deal here with probabilities and they have their own special rules. Let 
~^q(At\ (x,t)) be the probability that the system that's in x at t will not jump away from x in At. 
This probability, of course, is 

-^q(At\(x,t)) = l-q(At\(x,t)). (22) 

The probability that the system still will not jump in the next dt is ^q (At + dt\ (x, t)) and it can be 
decomposed into the product of the probability that the system will not jump in At and the probability 
that it will not jump in the following dt either: 

-nq (At + dt\ (x, t)) = ->q (At\ (x, t)) x -.g (dt\ (x, t + At)) 

= ->q (At\ (x, t)) (1 - q (dt\ (x, t + At))) (23) 
= ^q (At\ (x, t)) (1 - a (x, t + At) dt) , 



wherefrom 



This then integrates to 



d->q(At\Cx,i)) A . , 

— ,\ ,'; ' // = -a(x,t + At) dt. 24 



^q(At\(x,t)) =e -Jo a W x >*+*') dt '. (25) 
Now we drop the -i, reverting to q itself, which yields 

q(At\(x,t)) = l- e -J At *(x,t+t)dt\ (2g) 
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For the very small At w dt we should expect a (x, t) dt to be small too. The exponential function 
then will be approximated by 

l-a(x,t)dt (27) 

and (26) becomes (13), our starting point. We see here that the infinitesimal relationship (13) itself 
was not enough to reconstruct the correct relationship between q and a for the finite At. Additional 
information, provided in (23), was needed to get it right. 
Smoothness of a and w: In order to obtain smoothness conditions on a and w we need to inspect what is 
required of these two functions to ensure that the propagator density function II (x'|dt| (x,t)) satisfies 
the Chapman-Kolmogorov condition 

II (x'|d*| (*,*)) = 

f n ^„ ) U(x -x"\{l- X) dt\ (x + x",i + Xdt)) (28) 
U(x"\Xdt\(x,t)) dx". 

This condition assumes that the infinitesimal dt can be split into two sub-infinitesimals, [0, Xdt] and 
[A dt, dt] and that the process itself must be Markov-divisible on top of these two intervals. This is 
a weighty assumption, especially given that A is an arbitrary real number in [0,1]. It will result in 
further continuity conditions down the road. We begin by substituting (20) in (28). The first II under 
the integral becomes 

a(x + x",f+ Xdt) (1 - A) dtw(x -x"| (x + x", t + A dt)) 
+ (1 - a(x + x\t + Xdt) (1 - A) dt)6(x -x") . 

The second II is 

a (x, t) A dt w (x" | (x, t)) + (1 - a (x, t) A dt) 5 (x") . (30) 

We have to multiply them now, but we only keep terms linear in dt, because dt is infinitesimal after 
all. The first component in each sum is proportional to dt, so we can neglect first times first right 
away. But the second component of each sum contains a term that is not proportional to dt, which is 
the 6 term itself. In summary, we'll end up with the first component of the first sum times the delta 
from the second sum, plus the first component of the second sum time the delta from the first sum, 
plus a term that'll contain a product of the two deltas. This last term looks as follows 

5 (x - x") S (x") (1 - a (x + x",t + Xdt) (1 - A) dt -a(x,t)Xdt) (31) 

to which we add 



a (x + x" ,t+ Xdt) (1 - A) dtw(x -x"| (x + x", t + A dt)) 5 (x") 
+a (x, t) Xdtw (x" | (x, t)) 5 (x — x") . 



(32) 



This is to be integrated over f2 (x"). The deltas make the integration easy. S (x") simply converts x' 
to zero and S (x' — x") converts x" to x', which yields 



a(x,t + Xdt) (1 - A) dtw(x'\ (x,t + Xdt)) +a(x,t) Xdtw (x'| (x,i)) 
+ (1 - a (x, t + X dt) (1 - A) dt - a (x, t) X dt) S (x) . 



(33) 



We want this to be equal to 

a (x, t) dt w (x | (x, t)) + (1 - a (x, t) dt) S (x) . (34) 
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(37) 



The requirement imposes certain conditions on a and w. For example, comparing the 5 terms yields 

a (x, t + A dt) (1 - A) + a (x, t) A = a (x, t) . (35) 
This must hold in the dt — * limit. If we can assume that 

a (x, t + A dt) = a (x, t) + ^j^ A rff + ((A dt) 2 ) (36) 

then 

a (x, f + A dt) (1 - A) + a (x, t) A 
= (1 - A) (a (x, t) + ^^-Xdtj + Aa (x, t) + O ((A dt) 2 ) 
= a (x, i) + (1 - A) A + O ((A dt) 2 ) 

— > a(x, t). 

We add a similar assumption regarding u>, that is, 

w (x'| (x,t + Xdt)) = 
w (x' | (x, t)) + M*^')) Xdt + O ((A dt) 2 ) (38) 

to demonstrate similarly that 

a (x, t + A dt) (1 - A) to (x' | (x, t + A dt)) +a(x,t)\w (x | (x, t)) 
— » a(x, t) u> (x'l (x,t)) . 

But by now we can see it even without the explicit computation. If both a and w are to be expanded 
as per (36) and (38), the first term, the one that does not involve dt, is simply o(x, t) w (x'| (x,t)). 
The terms that are proportional to A cancel out. All other terms have dt in them, so they all vanish in 
the limit dt — ► 0. Consequently, we see that assumptions (36) and (38) are sufficient to ensure that the 
propagator (20) satisfies the Chapman- Kolmogorov condition (28). However, we have not proven that 
they are necessary. There may exist less restrictive conditions that would still ensure the satisfaction 
of (28). 

Functions a(x,t) and u> (x' | (x, t)) are called the characterizing functions of the jump Markov process. 
Whereas the continuous Markov process is also described by two functions (see module m44258, where 
we call them u.(x,t), the drift function, and D(x,i), the diffusion function), the functions depend on two 
variables x and t. Here, for jump processes, we have three variables. The third one is x'. The resulting 
description is going to be more elaborate. We will also show that in a certain limit jump processes become 
continuous processes. 

Looking at (20) we see that the two functions enter the expression for the propagator in a special way, 
namely through the product of a and w and then through a itself in the second summand, the one proportional 
to the Dirac delta. Because w (x'| (x,t)) is the probability density in x', it must integrate to 1. In turn, 
a(x, t) does not depend on x'. We can therefore define 

W(x'|(x,t)) =a(x,t) ™(x'|(x,t)), (40) 

for which we'll find that 

a(x,t)= / W(x\ (x,t)) dx (41) 
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and 

u>(x'|(x,i)) = T ^i X| i X '^l 3 r - (42) 

Function W (x'| (x, i)) is called the consolidated characterizing function of the jump Markov process and it 
can be used instead of a and w to encode the jump process propagator density as follows 

n(x\dt\(x,t)) = 

W(x|(x,t)) dt+ (l- (Sn^WWfct)) dx') dt) (5(2;'). 



The expression 



W (x'| (x,i)) cftdx' (44) 



is the probability that the system in x at t will jump in the next dt by between x and x + (fx away from 
x. 

Now, we switch to one dimention, x — > x and x — * x . 

Once we have the jump propagator density function, we can calculate propagator moments, namely 

IMM) dt = C oo (x') n U(x\dt\(x,t)) dx' 

= I-oo( x T ( a 0M) eftttf(x'|(a;,t)) + (l-a(a;,t) di) 6 (x)) dx' (45) 
= a(x,t)dt J^° 00 {x') n w (x'\(x,t))dx. 

Here we have made use of 



(x ) 5(x) dx =0 (46) 
to kill the second term, the one with 1 — a. From the above then 

/oo 
(x') n w (x\ (x,t)) dx'. (47) 
-oo 

Defining the n th moments of w and W, the consolidated characterizing function of the jump process, by 

/oo 
(V) w (x'\ (x, t)) dx' (48) 
-oo 

and 

W„{x,t) = (x') n a(x,t) w(x'\ (x,t)) dx 
= JZ (x') n W(x'\(x,t))dx', 



we can rewrite (47) as 



II„ (a;, t) = a (x, t) w n (x, t) = W n (x, t) . (50) 



For the above definitions to be meaningful, the corresponding integrals must be convergent: the moments 
n„ exist if the corresponding moments of w (or W) exist. 
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4 The Jump Kramers-Moyal Equations 

The Kramers-Moyal equations (forward and backward) were derived in module m44014 from the Kolmogorov 
equations. They allowed us to express the time derivatives of the Markov process probability density function 
P ((#, t) | (xq, to)) in terms of the propagator density moments, thus providing us with some sort of evolution- 
ary equations, not unlike the Schrddinger equation of quantum mechanics — though, in general, as we'll see 
in the case of the jump processes, they may not be explicitly solvable. In the case of the continuous Markov 
processes, the forward Kramers-Moyal equation turned out to be the familiar Fokker-Planck equation that 
with some additional non-Markovian assumptions could be converted into the Schrddinger equation — the 
"non-Markovian" phrase here being important. 
The forward Kramers-Moyal equation is 

-p {{x, t) 1 (s 0) i )) = E -^f- 0^ ( n ™ (*- *) p *) 1 ( x °> • ( 51 ) 

Substituting (50) yields 

gP((x,t)\(x Q ,to)) =E^i^^(o(«.H(i,t) P((x,t)\(x ,t ))) 

= En=l i= &&(Wn(x,t) P((x,t)\(x ,t ))). 
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